DOCUMENT RESUME 



ED 290 762 

AUTHOR 
TITLE 



PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



TM Oil 002 

Loesch-Grif f in , Deborah A.; Ringstaff, Cathy 

When Unintended Consequences Become the Main Effect: 

Evaluating the Development of a Foster Parent 

Training Program. 

Oct 86 

22p.; Paper presented at the Annual Conference of the 
American Evaluation Association (Kansas City, MO, 
October 29-November 1, 1986). The last page may not 
reproduce clearly. 

Reports - Evaluative/Feasibility (142) — 
Speeches/Conference Papers (150) 

MF01/PC01 Plus Postage. 

Child Advocacy; County Programs; Databases; 
Evaluation Methods; Financial Support; *Foster Care; 
Foster Children; *Foster Family; *Needs Assessment; 
Nonprofit Organizations; *Parent Counseling; *Parent 
Education; *Program Evaluation; Social Services; 
Social Work; Social Workers; Staff Development; 
Systems Analysts 
*Evaluator Characteristics 



ABSTRACT 

A program of education, training, and support 
provided to foster parents in a California county through a nonprofit 
agency is evaluated. The evaluators 1 experience indicates that: (1) 
evaluations are gaining in popularity; (2) role shifts by evaluators 
are sometimes difficult to perceive; (3) program staff are unlikely 
to use evaluative feedback unless it matches their own agendas; (4) 
communication breakdowns occur even in small organizations; and (5) 
the reorganization of existing services is sometimes preferable to 
development of a new program. Although direct measures of the 
program's services do not show significant improvement in tne status 
of foster children and foster parenting, it was found that the 
program does have an impact on the foster care community. 
Recommendations based on the experience of the evaluators include 
support of database development and maintenance, improvement of 
county services coordination, providing of funding for planning and 
development as well as implementation of a program, close scrutiny of 
goals of short-term programs, attention of unintended consequences of 
programs, and continued evaluation of field-based social services. 
(TJK) 
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Introduction 

In any evaluation project , evaluators will, in all likelihood, learn about the content 
of the program they are evaluating. In our situation-as beginning evaluators enrolled 
in a training program-the most important lessons we learned had absolutely nothing to 
do with the content of the program we were evaluating. For example, one might 
assume that we would learn a great deal about children at risk since we were 
evaluating a foster parent training program. Instead, what we learned focused mainly 
on the different types of roles evaluators must assume when faced with evaluating a 
developing program. To experienced evaluators, the lessons we will discuss in this 
paper are no doubt ones with which they are well acquainted. For novice evaluators, 
however, this paper may shed some light on the complex process of evaluating a 
program which is seated within a network of social services. Perhaps a discussion of 
this sort will forewarn novice evaluators about what to expect if they take on an 
evaluation project similar to ours. 

Background 

In 1983, a Grand Jury Report in a county in California pui forth several mandates 
regarding the care and assessment of foster or dependent children. Potential foster 
parents were the target of many of th3se mandates, including: 1) a system of mandated 
training for foster parents; 2) a network of support groups pairing veteran and new 
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foster parents in a buddy system; 3) recording attendance at orientation meetings, and 
4) assessing the potential for child abuse during the licensing of new foster homes. 
The assumption was that by helping foster parents, the benefits would somehow filter 
down to the dependent children placed in their homes. 

Prior to 1983, foster parent organizations existed in nearly every geographical 
district of the county. However, funding cuts led to lay-offs of permanent Department 
of Social Services (DSS) personnel associated with these foster parent organizations. 
As as result, only two fester parent organizations were active when we began the 
evaluation in April, 1985. One was specifically targeted for the needs of Fost-Adopt 
parents (those interested in adopting a shild) , the other served Hispanic foster parents. 

In addition to these two organizations, the only other loimal program that 
remained for meeting the specific needs of foster parents were the three orientation 
meetings sponsored by DSS. Attendance at these meetings was required of all 
individuals interested in adopting children or becoming foster parents. These didactic 
presentations focused mainly on generic problems parents might encounter with 
children rather than issues related exclusively to foster parenting. These three 
meetings were part ot the recruitment and licensing process and took place prior to 
parents having any placements in their homes. Hence, much of the discussion 
centered around licensing requirements. Since many individuals had still not received 
a foster child or fost-adopt placement six months after these orientation meetings, it is 
questionable whether any of the material presented to them could be recalled by these 
individuals. Thus, in response to a perceived need, a non-profit agency proposed and 
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was funded to offer a program of education, training, and support for foster parents. 

Enter Evaluators 

*Lesson # 1 : Evaluations are gaining In popularity. While few people seem to 
understand the complexities and benefits of evaluation research, they seem 
acquainted with the idea that it increases the credibility of service programs. 
Therefore, funding foundations are more frequently requesting that programs include 
evaluations as a requisite for funding. Unfortunately, evaluators, funding agencies, 
and agencies responsible for program delivery all seem to have different perspectives 
on evaluation processes and outcomes. 

An evaluation component was written into the proposal at the request of the 
funding agency. The evaluation began during the first days of the program's 
development, prior to the hiring of the social worker who was supposed to design and 
implement the program. The program was funded for one year. Other than the 
promise to deliver a program of education, training, and support, and a global sketch 
of the needs of foster children in the county, the proposal offered little in the way of 
specifics: they did not have a clear sense of what they wanted to do, nor what they 
wanted us to do. 

The role we would assume in relation to the program staff (N=2) was open for 
definition, as was the program they hoped to deliver. One role we consistently 
maintained was our role as ^ valuation research educators. We spent a good deal of 
time describing evaluation, distinguishing this activity from traditional conceptions of 
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pass/fail judgments of program efficacy. We also tried to clarify what activities would 
be inappropriate or hindrances to the evaluation process. When we first discussed 
our roles as evaluators, a prominent member of the advisory committee insisted that 
the best way to evaluate the program would be through an experimental design 
complete with randomized assignment. Ultimately, the project staff did concur that an 
experimental design would not be appropriate. Nevertheless, we never reached 
consensus on exactly what the evaluation should entail. 

In the beginning of the project we assumed that the funding agency would be 
interested in the kinds of information typically produced in an evaluation: data related 
to how the program influenced the participant^ data about the various program 
components that were most effective in meeting the program's goals, and how the 
program ultimately affected the foster children. Instead, when we submitted our 
six-month report, we found that the foundation seemed more interested in "'.e agency's 
ajcount of its expenditures than in our account of what had taken place over the 
six-month period. Moreover, the agency had hoped that our six-month report and our 
firal report could be used to convince the funding agency to allot more funds for the 
program. When the six-month report failed to meet this expectation, there was a 
dramatic shift in our relationship with the agency staff. After reflecting on this shift, we 
realized tnat throughout the evaluation there had been other, less dramatic, shifts in 
our role. 

* Lesson #2: Very often we don't recognize when our role shifts or how others 
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perceive us until we are asked something that crosses our own interna ' boundaries 
regarding where evaluation begins and ends. 

More often than not, the shift we experienced in roles coincided with a shift in staff 
goals, perceptions, or expectations. Sometimes this shift centered around a conflict 
between the staff and team cf evaluators which sharpened our own sense of what 
social service agencies conceive evaluation to be. 

Collaborator: When we first began the program evaluation we had the 
expectation that the program was clearly conceived, or at least had some semblance of 
coherence. That is, we expected that the staff held ( if not on paper, than in mind) a 
clear conception of an intact program-a model against which they would attempt to 
match and evaluate their own efforts. They had neither. What existed in reality were 
loosely coupled events which left the immediate impression of an unmet need in the 
community. Therefore, we joined forces with the director in searching the literature, 
conducting ERIC searches, reviewing documents from other programs, and referring 
possible candidates to the program director. 

Our review of the literature yielded little evidence about the few models after 
which this county agency might pattern its own program. According to law professor , 
Mike Wald, a great deal has been written about dependent children, but very little 
has been written about the effects of foster parent training or foster 
parenting. Our own literature search confirmed this. 

In the past decade, many programs have been developed to address the needs 
of foster parents. However, most of the information written on foster parenting in the 
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literature is descriptive in nature. Of the 33 listings obtained in the computer search, 
only 3 involved experimental or evaluation research studies of foster parent training 
programs. 

What could we conclude about foster parent training? Because so few 
studies have been documented it was, quite frankly, difficult to conclude anything 
about what works best in foster parent training programs. It was not even clear 
whether foster parent training programs are the most effective means of preparing 
adults to function successfully in this role. Yet, insufficient evidence was not grounds 
for abandoning the development of a foster parent training program. It is* however, the 
best reason available for including evaluation and research components in 
field-based social service programs. 

None of the tasks we performed in the role of collaborator are particularly 
unusual for evaluators to undertake. But these activities created In us a sense of 
investment in getting the program started. In essence, we lost what little ground we 
had gained in educating the agency about the purpose and role of evaluation. Luckily, 
even our role as collaborators had its limits. As researchers, we wanted to base our 
recommendations in the literature. We didn't feel comfortable advocating one structure 
over another, since little evidence existed to suggest that any particular method was 
superior to others. 

At this point the program director concluded that the outside resources were 
either "outdated" or inappropriate. She began to rely on anecdotal information from 
other county and state program directors and from members of her advisory board, 
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some of whom were foster parents. Since we did not want to use this as the sole 
source of information upon which to design the program, we intensified our evaluation 
efforts in two activities: networking (i.e., shared problem solving with other related 
agencies and social service providers) and needs assessment. It vas about this time 
that the social worker was hired, so the burden of finding, developing, and integrating 
program prototypes fell on her shoulders and we turned our attention to gathering 
specific information about the local needs of foster parents and children. This change 
in staff initiated the shift in our role from collaborator to informant. 

*Lesson # 3 : We found that program staff would not necessarily use the information we 
provided if it didn't match their own agendas , even if we made every attempt to deliver 
information which provided answers to the questions they themselves had formulated. 

Informant . It was this third role that moved us away from and out of the program's 
boundaries into the social system of support services for foster parents and children. 
Our relationships with these other agencies were prompted by a need to get 
information about the foster children and foster parents in order to feed it back to the 
agency as they began developing the foster parent training program. 

We wanted to avoid duplicating statistical data collection if others had been 
engaged in that activity as a part of delivering services to foster parents and children in 
the county. However, we found that the data in the statistical tables on dependent 
children produced and provided by DSS was out of date and difficult to interpret. We 
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were informed that approximately 30 children per month entered the foster care syslem 
in the county. Although an examination of the statistical data did not make clear where 
these 30 children ended up, we extrapolated from the representative categories what 
outcomes might result for these children once they entered the fester care syste.r, (sae 
Appendix A). One thing was clear from the data: foster children were at least alive, if 
not well, and living in the county. It appeared that there were equal numbers of 
children placed in group homes and single family homes. Since tha target audience of 
the program was singje. family foster parents, w3 needed to determine whether these 
individuals were in need of the type of program the agency was going to provide. 

As a means of gathering data to answer this question, we conducted mail and 
phone surveys and held interviews with key informants in the other social service 
agencies, and recorded information made available to foster parents throughout DSS" 
orientations. This process put us into contact with many people who were not directly 
involved with the program, it also fleshed out the initial picture sketched in the 
proposal by providing details which accentuated the needs of foster parents and foster 
children. The details we collected painted a different picture than the one the program 
director had anticipated. Need was not the only critical factor in getting an audience 
to partake of a service, even one that is specifically developed with their needs in mind. 
According to the vast majority of social service providers that we spoke with, 
attendance at meetings and program sessic.is for foster parents was .notoriously low . 
The program we were evaluating was no exception. 

The problem of getting warm bodies to a workshop, a training session, or support 
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group is not a new one. Yet, the question "Why don't people attend?" has never been 
answered to the satisfaction of social service programs and personnel. Therefore, they 
continue to develop new programs, or reinvent old programs with new names, in the 
belief that one will finally capture the attention, interest, and presence of the target 
audience. We believed foster parents in this cou.ity might reveal some new insight 
into why other programs had failed, and where this program might hope to succeed. , 
However, the type of information we sought to collect and disseminate became 
clouded by the expectations of the program staff who had been holding support groups 
for which they wanted specific information-information which we believed would 
violate the confidentiality of the participants and the social worker. 

The dilemma we faced was how to get information that was useful to the program 
staff, but that was also ethically appropriate to gather. The information we gathered 
ended up lying dormant under more pressing issues: getting people to attend the 
program that existed, and prodding the development of the program into more creative 
outlets than the stock avenues already deemed innappropriate b> the director. For 
example, in our needs assessment surveys we found that foster parents in this county 
were more interested in attending program sessions that provided training (i.e., specific 
information and techniques that would help them deal with the day-to-day problems of 
foster parenting). Instead of providing this service, the program social worker insisted 
that support groups (i.e., informal, unstructured gatherings where people share 
whatever is on their minds) should be the first step in program development. Thus we 
found that the information we had gathered was brushed aside. 
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Le sson #4: Even in small organizations there ca. be a complete breakdown in 
communication which ultimately affects the evaluators' role. The need to look at 
multiple sources of perceptions is crucial for defining and evaluating the program. 
Why do we look at? Which is the target audience the program is intended to impact? 

Mediator: In solving the dilemma raised in our roie as informant, we firmly 
planted ourselves back in the program's sphere of influence and into a new evaluation 
role as mediator. We discovered that the director and social worker were at odds in 
their goals and plans, and that the drive to deliver services was stronger than ever. 
Two conceptual frameworks regarding what form a foster parent training program 
should take were at work. These personnel conflicts led to further problems in defining 
our role as evaluators. 

By this time it was clear that we were already committed to evaluating the 
development of the p. jgram, rather than the program itself. When we were brought 
together with the program staff to assist them in defining their roles and goals, ve 
viewed this as another step in the evaluation of the process of shaping a particular 
form and style of social service delivery. 

At this time, we were also collecting data regarding the components of the 
program that wjarejn operation, which we intended to feedback to the staff as they 
progressed each week. We had been calling foster parents to find out if they were 
aware of the program's existence. When we were asked by the program director to 
inform them about upcoming program events, we once again recycled through our 
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educational role and discussed why this would not be appropriate if we v.i<shed to 
maintain a neutral stance in gathering data regarding participant perceptions and 
concerns. After making the declaration that we were not official program staff, we were 
once again free (i.e. excommunicated) to explore the program from afar. 

The shift in our role as evaluators coincided with a shift in the program's direction. 
Rather than just offering support groups, the agency held a conference with guest 
speakers knowledgeable about the foster care system and continued to hold these 
types of presentations (on a smaller scale) twice a month. Although the social worker 
and program director were anxious io find out how useful foster parents were finding 
these support groups, we began to doubt the utility of evaluating the impact of the 
program on the target audiences (i.e., foster parents and foster children) because the 
support sessions were so poorly attended by footer parents. 

It was at this time that we began looking at the unanticipated consequences of 
the program. It was clear from our observations and assessment of the conference and 
several of ths subsequent meetings that the program was having an impact on the 
network of foster care providers, of which the foster parents were but one part. 

Lesson # 5 : Just because people say there's a need and the target group affirms that 
services are lacking does not necessan'ly justify the development of a new program, it 
may mean that a reorganization of existing services is needed. 

Systems Analysts: Eariy on in the evaluation we had identified three audiences 
which were critical to the development and success of the program. They included 
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foster parents and children, the community of helpers, such as caseworkers, probation 
officers, community agency workers, school counselors, and family doctors, and finally, 
those charged with decision making and power over the conduct and welfare of the 
first two audiences-the juvenile court system, the administrators and supervisors of 
DSS, and child advocacy organizations. While we had made initial contacts with this 
last audience during the needs assessment and networking phase, our primary efforts 
had concentrated on gathering information f r om the first two audiences. 

After twice shifting to and from the program and the social system of support 
services, we ended the evaluation by defining the importance of the program, its 
components, and its effects from a global, external perspective. We found that the 
program was both helped and hindered by their reliance on the existing social service 
network. They were hindered because the network that existed was poorly structured 
and relied on inadequate information about the target audiences. However, the 
network did allow them to contact resource specialists who were knowledgeable about 
the needs of foster parents and existing services of which they could avail themselves. 
!n turn, the program we evaluated contributed to a stronger networking system by 
improving the communication between different social service providers. 

Conclusions 

Although originally everyone in the foster care community, including social 
workers, juvenile court judges., child advocates, and the foster parents themselves 
expressed the bt..af that a foster parent program was a necessary service, it is difficult 
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to assess whether or not this program was successful in creating a "client group" who 
would demand that the services be continued with public support. Possibly, given 
more time, this program might have expanded the participating core of foster parents 
into a more powerful and cohesive group willing to fight for their rights to more 
adequate services. 

Although direct measurements of the progrrm's services did not indicate 
significant improvements in the status of foster children and foster parenting, the 
program did have an impact on the foster care community. The initial bases for 
assessing the needs and recruiting program participants (e.g., networking), became 
the most profound development and outcome of the program's year-long efforts. In our 
final interviews with key informants from the decision making and powerbase bodies of 
the foster care community, all commented upon the renewed vision and opened 
channels of communication among the various agencies. Moreover, channels of 
communication which heretofore did not exist were constructed just as others which 
had broken down were reopened. The program never developed to its fullest 
potential, but in the process it served as a catalyst for new program funding for foster 
children and as a forum for clarifying and prioritizing the needs of foster parents. Thus, 
the unintended consequences of the program became the main effect. 
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Summary and Recommendations 

We would like to conclude with some of the lessons we have learned from the 
perspective of defining our role as evaluators and in relation to evaluating a foster 
parent training program which is embedded in a social service system. The most 
critical lesson learned is that it is extremely important for evaluators to maintain a 
respectable distance when they are called in to conduct an evaluation for a small 
organization, especially when the program they are evaluating is also being "birthed". 
The second critical lesson involves one which plagues many social service agencies 
receiving funding to "change the world" in less time than it takes to figure out what 
needs changing. Particularly, in evaluating program development it is important to get 
people to commit themselves to a plan of action so that the first year doesn't get 
bogged down in rehashing old schemes or hit-and-miss efforts. Finally, when called in 
to evaluate a program in the development phase, ask yourself: 

1) Who said there was a need for a new service? 

2) How much do we know about this need? 

3) What services already exist for this group? 

4) Why are existing services falling short of meeting this need? 

5) What might be done to improve or reorganize services that already exist? 

Faced with developing a one-year program for the training, education, and 
support for foster parents in the county, the agency was faced with a number of 
obstacles that will undoubtedly plague other social service organizations interested in 
developing programs for this target audience : 

1. Lack of a reliable and accessible data base about foster children and foster parents 
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from which to draw information about the characteristics and needs of their target 
audience. 

2. Lack of literature about foster parent training programs from which to draw 
information for planning an effective program. 

3. Lack of coordination between county services aimed at meeting needs of foster 
parents and dependent children. 

4. Lack of time for sufficient planning and development for the program since it was 
funded for one year during which time it was expected to provide services. 

5. Overextended foster parents who were, for the most part, unwilling to devote time to 
a program even when they believed the services would be valuable. 

Despite these obstacles, this program achieved the following: 

1. Dissemination of information regarding available services to a large group of foster 
parents and other community agencies. 

2. Increased communication and coordination between individuals and agencies 
responsible for services for foster children and foster parents. 

3. Development of a new program specifically for foster children, which grew out of the 
older program. 

4. Increased public awareness about the plight of foster children due to dissemination 
of information about the program through the media and through the efforts at 
networking with other agencies providing social services. 

5. Recruitment of a "core" group of foster parents who continued to show interest in 
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future developments at the close of the first year . 

Based on our experiences as evaluators of this program, we offer the following 
recommendations to any agency who is attempting to develop social services for the 
foster care community: 

1 Continued efforts to develop a reliable and accessible data base in the county on 
both foster parents and foster children. Without this data, it is doubtful whether any 
program can adequately plan to meet the needs of their target audience. 

2 Ircreased efforts at coordinating county services, both, public and private, that 
attempt to serve the needs of the foster csre community. 

possibly by establishing a central clearinghouse or hotline that is accessible to foster 
parents. 

3 Funding should be provid ed for the plannin g and development of a program not 
just its implementation. Feeling pressured to somehow produce tangible outcomes in 
a short period of time can lead to the initiation of a program that is not adequately 
planned. Expected "start-up" time should be explicitly stated in proposals for funding 
so that both funding agencies and those responsible for implementing the program do 
not have unrealistic expectations. 

4 Programs that are funded for only one yeqr need to more dearly deviate how thev 
will achieve their goals, and to realize that such programs can only have a very limited 
impact due to their short duration. 

5 Those who fund p-^ogals implement programs or evaluate social programs need 
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to pay close a ttention to the unanticipated consequences of programs in the case of 
this agency's program, these unintended consequences were in essence the main 
effects of the program. 

6 Since the data base on foster parenting is limited, we believe in the need for 
continued evaluation of field-based socia l services aimed at improving thp f og tpr rgrp 
system. These evaluations should be completed by "outsiders" to the system, because 
those who are somehow invested in the system will find it difficult to remain unbiased. 
Since funding for all social services is limited, agencies who finance such programs 
have the responsibilty both to themselves and the clients of the programs to make sure 
that the impacts of these programs are evaluated. We applaud the fact that such an 
evaluation component was requested by the funding foundation, and that the program 
staff were most cooperative in aiding our efforts. 
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evaluation? 
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#3: Utility of information 
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#4: Examining multiple 
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success 

#5: Determining the need 
for new services 
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